feat(storage): Postgres daily partitioning + DROP-PARTITION retention#53
Merged
Conversation
Adds an opt-in Postgres declarative-partitioning adapter for the `logs`
table, gated by `DB_POSTGRES_PARTITIONING=daily` (default off). When
enabled, `logs` is provisioned as a `RANGE PARTITION BY (timestamp)`
parent with the composite PK `(id, timestamp)`; AutoMigrate skips the
model and a `PartitionScheduler` runs hourly to (a) ensure today +
lookahead future daily partitions exist and (b) DROP partitions whose
range predates the retention cutoff. DROP PARTITION beats row-level
DELETE for retention by orders of magnitude.
`RetentionScheduler` reads `repo.LogsPartitioned()` and skips the
`logs` DELETE branch when partitioning is active so we never run two
retention paths against the same table.
Greenfield only — startup refuses to enable partitioning if `logs`
already exists as a non-partitioned table. Migrating an unpartitioned
`logs` to partitioned requires data movement and is out of scope.
`pg_trgm` GIN indexes are declared on the parent and propagate
automatically to current/future child partitions (Postgres ≥ 11).
New telemetry: `otelcontext_partitions_dropped_total` (counter) and
`otelcontext_partitions_active` (gauge). New env vars validated at
config load: `DB_POSTGRES_PARTITIONING` ("", none, daily) and
`DB_PARTITION_LOOKAHEAD_DAYS` (0..365). Daily mode is rejected when
DB_DRIVER != postgres.
Tests: 6 Postgres integration tests (testcontainers, auto-skipped
without docker) covering the partitioned schema, child-partition
routing, DROP_EXPIRED, greenfield guard, scheduler lifecycle, and
pg_trgm GIN inheritance. 4 unit tests for the date/identifier helpers.
Docs updated in CLAUDE.md and OPERATIONS.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
logstable, gated byDB_POSTGRES_PARTITIONING=daily. Default remains the legacy unpartitioned schema.PartitionSchedulerruns hourly: ensures today +DB_PARTITION_LOOKAHEAD_DAYSfuture daily partitions exist, drops partitions whose entire range predates the retention cutoff (DROP TABLE— orders of magnitude faster than DELETE).RetentionSchedulerreadsrepo.LogsPartitioned()and skips thelogsDELETE branch when partitioning is active.tracesandmetric_bucketscontinue to use the existing batched DELETE path.pg_trgmGIN indexes are declared on the parent and propagate automatically to current/future child partitions (Postgres ≥ 11).otelcontext_partitions_dropped_totalandotelcontext_partitions_active.Why
Phase 3b + Phase 5 of the 7-day-retention robustness initiative. Row-level DELETE at 7-day × 100–200 services becomes the dominant DB cost at scale; DROP PARTITION turns retention into an ~instant DDL. Postgres-only and opt-in per the board ruling — SQLite remains the zero-config default.
Greenfield only
Startup refuses to enable partitioning if
logsalready exists as a non-partitioned table. Migrating an unpartitionedlogsto partitioned requires data movement and is out of scope for this phase. The greenfield guard is enforced and tested.Test plan
go test ./... -race -count=1— full unit suite greensudo go test -tags=integration ./internal/storage/ -run TestPGPartition— 6/6 partitioning integration tests pass against postgres:16-alpine via testcontainersTestPGPartition_LogsTableIsPartitioned— verifies relkind=p and ≥5 initial partitionsTestPGPartition_InsertRoutesToCorrectChild— verifies row-routing to today's partitionTestPGPartition_DropExpired— verifies expired partition is dropped, today's staysTestPGPartition_GreenfieldGuard— verifies refusal on existing non-partitionedlogsTestPGPartition_SchedulerDropsExpiredAndCreatesLookahead— full scheduler lifecycleTestPGPartition_PgTrgmIndexesPropagateToChildren— verifies GIN inheritanceTestPG_ILIKE_CaseInsensitiveSearch,TestPG_Bytea_CompressedTextRoundTrip,TestPG_VacuumAnalyze_OutsideTx— pre-existing PG paths still passpartitionNameForDay,quoteIdent,parsePartitionUppergo vet ./...andgolangci-lint run --new-from-rev=origin/main— clean🤖 Generated with Claude Code